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Executive  Summary 

STANDARDIZING  SUBMISSIONS  TO  THE  MAJOR  AUTOMATED 
INFORMATION  SYSTEM  REVIEW  COUNCIL 


The  Office  of  the  Assistant  Secretary  of  Defense  for  Production  and  Logistics 
(P&L)  sponsors  the  development  and  implementation  of  automated  Logistics 
information  systems  used  by  the  Military  Services  and  Defense  agencies  to  satisfy 
their  mission  needs  and  to  increase  their  management  efficiency.  If  a  system  is  large 
enough,  P&L  must  submit  documentation  on  it  to  the  Major  Automated  Information 
System  Review  Council  (MAISRC),  which  seeks  to  ensure  that  costs  are  reasonable 
and  effective  management  is  in  place.  The  MAISRC  requires  rigorous  cost  and 
benefit  analyses  of  the  information  system  under  review  and  of  system  alternatives. 
The  analysis,  documentation,  and  MAISRC  approval  process  frequently  delays 
system  development. 

We  examined  the  procedures  P&L  uses  in  preparing  submissions  to  the 
MAISRC  and  identified  benefits,  costs,  and  organizational  interfaces  as  three  broad 
areas  in  which  standardization  may  benefit  P&L-sponsored  submissions.  Benefits  as 
presented  in  MAISRC  submissions  tend  to  be  understated,  principally  because 
systems  are  frequently  justified  on  the  basis  of  quantifiable,  economic  criteria.  The 
many  unquantifiable  system  benefits  are  more  difficult  to  analyze  and  consequently 
are  often  understated.  Program  managers  must  analyze  those  system  benefits  more 
closely,  particularly  in  an  environment  of  constrained  funding  or  cost  growth.  They 
should  also  define  operating  baselines  and  implement  standardized  procedures  to 
track  both  benefits  and  costs. 

In  preparing  MAISRC  submissions,  P&L  needs  to  ensure  that  the  cost  structure 
of  the  proposed  information  system  is  consistent  with  its  capability.  All  capabilities 
should  have  corresponding  cost  categories.  Analysts  must  identify  those  costs  and 
tabulate  them  across  the  full  system  life  cycle.  P&L  should  encourage  the  use  of 
parametric  costing  methods  at  an  early  stage  to  supplement  analogy-based 
procedures  and  to  validate  engineering  estimates  and  vendor  quotes.  Particular 
emphasis  should  be  placed  on  software  life-cycle  costing.  Consistency  in  presentation 
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format  with  guidance  provided  by  the  Office  of  the  Assistant  Secretary  of  Defense  for 
Program  Analysis  and  Evaluation  (PA&E)  is  desirable  although  as  a  practical 
matter  such  consistency  may  not  be  attainable. 

Organizationally,  P&L  should  take  the  following  steps  to  expedite  the  MAISRC 
process: 

•  Articulate  system  requirements  precisely,  and  structure  the  proposed  cost 
analysis  to  reflect  those  requirements. 

•  Clearly  document  the  pi  ogram  costs  and  benefits,  and  indicate  now  each  was 
measured. 

•  Communicate  early  and  frequently  with  other  organizations  in  the  MAISRC 
process,  such  as  PA&E  and  the  Office  of  the  Director,  Operational  Test  and 
Evaluation. 

To  address  its  long-term  needs,  P&L  needs  to  create  databases  of  historical  costs 
for  the  automated  information  systems  it  sponsors  to  take  advantage  of 
state-of-the-art  costing  methods.  Special  problem  areas  for  those  systems,  such  as 
software  development  and  maintenance,  should  be  given  first  priority. 


IV 


CONTENTS 


Page 


Executive  Summary  .  iii 

Chapter  1.  Introduction  .  1-1 

Definition  and  Purpose  .  1-1 

Organization  of  the  Report  .  1-1 

Chapter  2.  Estimating  Automated  Information 

System  Costs  .  2-1 

Background  .  2-1 

Identifying  Relevant  Cost  Elements  .  2-1 

Costs  and  Cost-Causative  Factors  .  2-2 

Selecting  Appropriate  Costing  Methods .  2-3 

Chapter  3.  Determining  Automated  Information 

System  Benefits  .  3-1 

Analysis  of  Benefits  .  3-1 

Identifying  Quantifiable  Benefits  .  3-2 

Identifying  Nonquantifiable  Benefits .  3-2 

The  Role  of  Baseline  Documentation  in  Improving 
Estimates  .  3-3 

Chapter  4.  Organizational  Issues  .  4-1 

Coordination  with  Other  Organizations  .  4-1 

PA&E  Perspective  .  4-1 

DOT&E  Perspective  .  4-3 

Chapter  5.  Conclusions  and  Recommendations  .  5-1 

Bibliography  .  Biblio.  1 

Appendix  A.  PA&E  Life-Cycle-Cost  Elements  .  A-l-A-3 


Appendix  B.  Cost  and  Benefit  Analysis  OASD(PA&E)-Recommended 

Formats  .  B-l-B-8 


V 


CHAPTER  1 


INTRODUCTION 


DEFINITION  AUD  PURPOSE 

The  Major  Automated  Information  System  Review  Council  (MAISRC)  reviews 
the  cost  and  management  that  the  Office  of  the  Assistant  Secretary  of  Defense  for 
Production  and  Logistics  (P&L)  proposes  for  the  development  and  implementation  of 
large  automated  information  systems  (AISs)  for  logistics.  The  MAISRC  review 
process  is  extensive  and  has  led  to  frequent  delays  in  AIS  development. 

To  expedite  the  MAISRC  process,  P&L  needs  to  understand  the  factors  that 
contribute  to  the  delays.  While  all  aspects  of  the  MAISRC  review  process  require 
comprehensive  documentation,  life-cycle  cost  estimates  and  cost-benefit  analyses 
have  historically  warranted  special  scrutiny,  and  that  scrutiny  is  frequently  the 
major  time-consuming  activity.  Since  neither  the  services  nor  P&L  have  a 
standardized  methodology  to  prepare  cost  and  benefit  estimates  for  AISs,  reviews  of 
cost  analyses  are  conducted  on  a  case-by-case  basis. 

In  this  report,  we  document  the  results  of  preliminary  analyses  of  MAISRC 
materials  and  processes.  These  analyses  are  based  primarily  on  information 
gathered  in  performing  cost  and  benefits  validation  for  two  Air  Force  AISs  [Weapons 
System  Management  Information  System  (WSMIS)  and  the  Engineering  Data 
Computer-Aided  Retrieval  System  (EDCARS)],  detailed  discussions  with  analysts 
and  managers  who  have  prepared  MAISRC  submissions,  and  interviews  conducted 
with  other  participants  in  the  MAISRC  process. 

ORGANIZATION  OF  THE  REPORT 

This  report  describes  our  observations  after  reviewing  MAISRC  cost  and  benefit 
estimates.  It  also  describes  organizational  concerns  that  should  be  addressed  in 
preparing  MAISRC  submissions  and  outlines  steps  that  will  ensure  satisfactory 
tracking  of  costs  and  benefits.  The  results  of  the  analysis  are  presented  in 
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Chapters  2,  3,  and  4.  Conclusions  are  provided  in  Chapter  5,  along  with 
recommendations  to  improve  the  efficiency  of  the  MAISRC  cost-benefit  analysis 
process. 
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CHAPTER  2 


ESTIMATING  AUTOMATED  INFORMATION  SYSTEM  COSTS 

BACKGROUND 

The  preparation  of  cost  estimates  for  MAISRC  submissions  is  a  time-consuming 
and  laborious  process.  Frequently,  DoD  Components  submitting  MAISRC 
documentation  devote  several  man-months  of  effort  to  preparing  estimates  in  formats 
required  internally,  only  to  be  compelled  to  repeat  or  revise  the  process  in  subsequent 
iterations  as  the  submission  is  subjected  to  review  at  various  levels. 

While  some  of  these  problems  are  due  primarily  to  differing  organizational 
requirements,  technical  issues  involved  in  AIS  costing  also  contribute  to  the 
problem.  Those  technical  issues  can  be  segregated  into  three  general  categories: 

•  Identification  of  relevant  cost  elements 

•  Establishment  of  relationships  between  costs  and  cost-causative  activities 

•  Selection  of  the  most  appropriate  costing  methodology. 

In  this  chapter,  we  describe  the  key  issues  that  MAISRC  submissions  should 
address  in  each  of  these  categories  and  identify  functional  areas  of  AISs  that  have 
proven  most  difficult  for  the  systems  we  examined. 

IDENTIFYING  RELEVANT  COST  ELEMENTS 

The  Office  of  the  Assistant  Secretary  of  Defense  for  Program  Analysis  and 
Evaluation  (PA&E)  has  identified  a  range  of  cost  elements  that  can  be  used  as  a 
guide  in  preparing  MAISRC  submissions.  (We  show  those  cost  elements  in 
Appendix  A.)  The  PA&E  guidelines  are  sufficiently  broad  to  address  general  AIS 
structures  and  do  not  need  amplification  to  be  used  as  a  point  of  departure  for  AIS 
analysis.  However,  they  are  merely  guidelines,  and  hence  they  are  not  definitive  for 
individual  AISs.  Each  component  submitting  MAISRC  materials  tailors  its  efforts  to 
the  specific  system  under  consideration.  Consequently,  any  cost  submission  uses  the 
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guidelines  as  an  initial  framework,  but  the  final  product  reflects  the  characteristics 
of  the  specific  AIS  under  consideration  and  the  development  phase  of  the  system. 

The  PA&E  guidelines1  also  suggest  certain  formats  for  the  presentation  of  costs 
(and  benefits)  within  this  framework.  These  formats  are  designed  to  ensure  logical 
consistency  and  completeness  and  to  facilitate  comparison  between  current  and 
previous  MAISRC  submissions.  (Examples  of  these  formats  are  in  Appendix  B.) 

ESTABLISHING  COSTS  AND  COST-CAUSATIVE  FACTORS 

Establishing  valid  causal  relationships  between  costs  and  cost-causative  factors 
appears  to  be  the  principal  difficulty  encountered  in  preparing  the  cost-benefit 
analyses  of  AISs  for  the  MAISRC.  Major  problems  in  the  MAISRC  cost  submissions 
frequently  center  on  whether  the  proper  and  relevant  costs  have  been  estimated ,  rather 
than  whether  the  costs  have  been  estimated  properly. 

There  are  two  separate  aspects  to  the  problem  of  estimating  costs  by 
establishing  causal  relationships.  First,  for  every  function  the  AIS  is  projected  to 
perform,  some  costs  must  be  incurred  somewhere.  Second,  all  components  of  these 
costs  need  to  be  recognized  over  the  life  cycle  of  the  AIS. 

The  first  c?  these  problem  areas  implies  the  need  to  reason  through  all  of  the 
inputs  needed  to  produce  a  desired  result,  recalling  the  axiom,  "You  don’t  get 
something  for  nothing.”  Although  the  need  for  such  thorough  reasoning  may  appear 
obvious,  it  can  readily  be  overlooked  in  AIS  costing,  particularly  since  requirements 
change  over  time  and  new  system  capabilities  are  added  incrementally. 

A  major  problem  cited  by  PA&E  is  the  failure  to  update  functional  descriptions 
after  Milestone  I.  That  failure  in  turn  leads  to  unsubstantiated  capabilities  that  are 
not  supported  by  the  original  cost  analysis.  Costing  problems  of  this  nature  can  be 
alleviated  by  reevaluating  and  updating  both  functional  descriptions  and  cost  models 
periodically,  ensuring  consistency  between  the  two. 

While  the  failure  to  update  functional  descriptions  over  the  AIS’s  life  cycle  is 
frequently  symptomatic  of  an  unstable,  evolving  requirement,  a  similar  problem  may 
also  occur  as  additional  potential  users  are  identified.  For  example,  EDCARS  was 

1  "Department  of  Defense  Automated  Information  System  Life  Cycle  Cost  and  Benefits 
Estimation  Guide  (DoD  Cost/Benefits  Guide),”  Office  of  the  Assistant  Secretary  cf  Defense,  Program 
Analysis  and  Evaluation,  30  May  1989. 
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projected  to  provide  engineering  data  to  assist  users  in  retrieving  information  for  bid 
set  preparation,  which  it  does.  However,  as  additional  functions  became  apparent  for 
EDCARS,  such  as  assistance  in  maintenance  activities  and  supporting  the  newly 
created  Office  of  Competition  Advocacy,  costs  associated  with  those  functions  needed 
to  be  explicitly  recognized. 

The  second  problem  area  involves  recognizing  the  costs  that  must  be  incurred  to 
maintain  a  capability  as  well  as  those  that  must  be  incurred  to  provide  that 
capability.  In  analyzing  such  costs,  the  analyst  must  ensure  the  following: 

•  All  categories  of  costs  have  been  considered. 

•  All  the  steps  necessary  to  meet  program  objectives  have  been  delineated. 

•  All  phases  of  the  system  life  cycle  have  been  addressed. 

Failure  to  recognize  all  costs  frequently  manifests  itself  in  MAISRC 
submissions  as  an  insufficient  life-cycle  cost  estimate.  Common  errors  include 
estimating  costs  only  for  a  specific  Program  Objective  Memorandum  (POM)  or  budget 
cycle  or  truncating  the  analysis  at  the  projected  date  of  full  operational  capability 
(FOC).  The  AIS  program  standard  life  cycle  includes  a  period  of  10  years  beyond 
FOC,  and  generally  includes  an  overall  system  upgrade  between  4  and  6  years  after 
FOC  to  respond  to  anticipated  advances  in  technology.  By  ending  the  analysis 
prematurely,  the  analyst  does  not  consider  the  full  range  of  operating  and  support 
costs,  system  upgrades  to  prevent  obsolescence,  or  preplanned  product  improvements. 

SELECTING  APPROPRIATE  COSTING  METHODS 

In  selecting  a  methodology  for  estimating  AIS  costs,  the  analyst  is  constrained 
by  three  key  factors: 

•  The  milestone  for  which  the  analysis  is  being  prepared 

•  Available  data 

•  The  resolution  required  in  the  estimate. 

Within  the  general  cost  structure  presented  in  Appendix  A,  the  analyst  has 
wide  latitude  to  select  acceptable  costing  tools  and  techniques.  Techniques  may 
include  analogy-based  estimates,  standard  engineering  estimates  and  vendor  quotes, 
and  parametric  cost  estimating  relationships. 
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Analogy-based  estimates  are  most  appropriate  for  the  early  milestones  and  in 
circumstances  in  which  historical  databases  are  not  available  to  use  in  projecting 
empirical  relationships.  These  estimates  involve  expert  judgments  and  experience  to 
gauge  the  degree  of  similarity  between  the  elements  of  a  proposed  system  and  known 
costs  of  previous  systems.  They  also  require  judgment  to  assess  how  any 
dissimilarities  should  affect  the  ultimate  cost  estimate.  As  the  system  progresses 
toward  later  milestones,  analogy-based  estimates  should  gradually  be  replaced  by 
firmer  estimates  derived  from  engineering  estimates. 

Engineering  estimates  are  built  up  from  component  cost  estimates  when  the 
components  that  most  strongly  affect  significant  cost  elements  are  reasonably  well 
known.  Such  estimates  may  be  provided  from  build-ups  of  hardware  costs  from 
standard  sources  such  as  Datapro,  or  from  quotes  solicited  directly  from  hardware 
and  software  vendors.  The  estimates  and  vendor  quotes  are  then  supplemented  by 
assessments  of  required  labor  hours  at  known  labor  rates.  Alternatively,  cost 
estimation  packages  are  commercially  available,  but  they  require  that  input  data  be 
specified  by  the  cost  analyst.  MAISRC  submissions  should  increasingly  rely  on 
engineering  estimates  as  they  progress  to  the  later  milestones  since  such  estimates 
provide  greater  resolution  as  the  program  becomes  better  defined. 

Parametric  cost  estimating  relationships  (CERs)  can  provide  greater  accuracy 
than  analogy-based  estimates  in  the  early  stages  of  a  program  even  though 
parametric  models  require  an  historical  database  for  development  and  calibration. 
By  using  parametric  CERs,  the  analyst  can  identify  prominent  cost  "drivers”  early  in 
the  estimating  process  and  is  better  able  to  assess  the  sensitivity  of  the  estimate  to 
key  parameters  and  to  analyze  the  impact  of  uncertainty  on  the  cost  estimate. 

Parametric  modeling  is  currently  used  widely  in  software  cost  estimating.  Of 
the  systems  we  examined,  software  development  cost  was  reported  to  be  the  single 
most  difficult  cost  element  to  estimate.  Although  software  costs  were  invariably 
estimated  using  commercially  available,  reputable  models,  input  parameters 
developed  for  the  models  led  to  considerable  disagreement.  Consequently,  estimates 
for  software  development  costs  ranged  from  $8  to  $150  per  line  of  code. 

Such  disparate  estimates  illustrate  the  state  of  the  art  in  software  cost 
estimation.  From  the  perspective  of  the  MAISRC,  the  use  of  parametric  models  to 
estimate  software  costs  greatly  enhanced  the  productivity  of  the  process.  By  enabling 
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the  participants  in  the  process  to  focus  their  discussion  on  differences  of  opinion  in 
key  input  variables  driving  the  cost  estimate,  parametric  models  narrowed 
discussion  to  a  manageable  level. 
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CHAPTER  3 


DETERMi  JNG  AUTOMATED  INFORMATION  SYSTEM  BENEFITS 

ANALYSIS  OF  BENEFITS 

The  DoD  Cost/Benefits  Guide1  prepared  by  OASD(PA&E)  suggests  methods  to 
identify  benefits,  measure  quantifiable  benefits,  and  evaluate  nonquantifiable 
benefits.  The  methods  are  presented  in  a  level  of  detail  similar  to  the  guidelines 
provided  for  cost  analysis:  broad  guidance  with  wide  latitude  for  analysts  to  select 
specific  techniques  appropriate  to  their  particular  circumstances. 

As  part  of  the  guidance,  the  DoD  Cost/Benefits  Guide  defines  "quantifiable”  and 
"nonquantifiable”  to  assist  cost  analysts  in  discriminating  among  costs.  A 
quantifiable  benefit  is  any  advantage  provided  by  the  AIS  that  can  be  valued  in 
monetary  terms  or  equivalents,  such  as  labor.  Conversely,  nonquantifiable  benefits 
are  defined  as  favorable  results  of  using  an  AIS  that  cannot,  by  nature,  be  valued  in 
monetary  terms.  The  analyses  we  reviewed  generally  categorized  benefits  as 
quantifiable  and  nonquantifiable,  although  the  level  of  supporting  documentation 
varied  widely. 

Irrespective  of  how  AIS  benefits  were  structured,  we  found  two  major  recurring 
problems  with  the  analysis  of  benefits.  First,  the  definition  of  the  program  baseline  is 
usually  inadequate  and  that  inadequacy  complicates  the  analysis  of  improvements 
brought  about  by  implementing  the  information  system.  Second,  insufficient 
procedures  are  available  to  determine  whether  tne  benefits  are  obtained  once  the 
system  h  3  achieved  initial  or  full  operational  capability.  These  problems 
overshadowed  the  purely  technical  issues  surrounding  identification  and  estimation 
of  AIS  benefits. 

The  remainder  of  this  chapter  addresses  areas  that  should  be  considered  in 
benefits  analysis  and  then  discusses  the  impact  that  improved  documentation  and 
tracking  systems  may  have  on  MAISRC  analyses. 


Ubid. 
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IDENTIFYING  QUANTIFIABLE  BENEFITS 

Quantifiable  benefits  are  those  that  either  reduce  costs  or  enhance  value. 
Cost-reduction  benefits  result  from  an  improvement  to  existing  operations,  while 
value  enhancement  benefits  provide  additional  capability  for  an  organization.  The 
OASD(PA&E)  DoD  Cost/Benefits  Guide  discusses  ways  to  measure  cost-reduction 
benefits  and  provides  general  comments  on  determining  measurable  quantities  in  an 
AIS  benefits  evaluation.  However,  it  does  not  offer  any  guidance  for  measuring  the 
value  of  new  or  enhanced  capabilities. 

To  identify  the  benefits  that  can  be  measured,  the  analyst  must  recognize 
operations  that  replace  those  of  the  baseline  system.  Replacing  a  particular  task, 
function,  or  piece  of  equipment  is  a  common  benefit  that  can  readily  be  converted  into 
dollars.  Replacement  operations  provide  substantial  value  to  the  quantifiable 
benefits  total  although,  again,  no  standardized  methods  for  quantifying  such  benefits 
are  available.  Finally,  cost  avoidance  is  an  area  that  can  be  quantified  by  forecasting 
the  increased  workload  that  will  be  avoided  with  the  implementation  of  the  AIS. 

IDENTIFYING  NONQUANTIFIABLE  BENEFITS 

Identifying  and  evaluating  nonquantifiable  benefits  offers  the  greatest 
challenge  —  and  is  perhaps  the  single  most  productive  area  for  analytical 
improvement  —  in  the  MAISRC  costing  process.  Theories  abound  on  how  best  to 
evaluate  nonquantifiable  benefits.  None  is  applicable  in  all  circumstances,  and 
virtually  all  have  some  flaws.  The  DoD  Cost/Benefits  Guide  suggests  a  few 
worthwhile  techniques,  although  several  others  are  acceptable.  The  guide  suggests 
benefits  be  categorized  and  then  compared  with  benefits  of  alternative  AISs.  The 
guide  acknowledges  that,  at  times,  a  narrative  description  of  the  characteristics  of 
the  nonquantifiable  benefits  may  be  the  best  that  the  analyst  can  provide.  The 
following  are  examples  of  nonquantifiable  benefits: 

•  Improved  decision  making 

•  Better  management  information 

•  Greater  versatility  or  flexibility 

•  Better  presentation  of  information 

•  Improved  timel  iness  of  information 
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•  Improved  staff  morale 

•  Fulfillment  of  operating  requirements. 

We  suggest  additional  types  of  intangible  benefits  that  are  directly  applicable  to 
DoD  organizations.  They  include  credibility  and  prestige;  information  integrity, 
flexibility  and  accessibility;  staff  training  in  new  technologies;  support  of  the 
planning,  programming,  and  budgeting  process;  and  better  relations  with  other 
Government  organizations. 

The  PA&E  guide  proposes  several  methods  for  measuring  nonquantifiable 
benefits,  ranging  from  a  simple  enumeration  of  the  benefits  to  a  cardinal  or  ordinal 
ranking  of  them.  Other  evaluation  methods,  including  multiattribute  decision 
theory  and  the  analytical  hierarchy  process,  are  appropriate  for  including  the 
judgments  of  more  than  one  individual. 

Numerous  other  analytical  techniques  such  as  the  following  may  be  used  to 
supplement  the  analysis  of  nonquantifiable  benefits: 

•  Delphi  technique 

•  Incremental  analysis 

•  Value  analysis 

•  Excess  tangible  cost 

•  Worst  case/most  likely  case/best  case  methods 

•  Checklists 

•  Critical  success  factors 

•  Cost-value  analysis. 

Information  on  these  methods  is  given  in  the  documents  cited  in  the 
bibliography. 

THE  ROLE  OF  BASELINE  DOCUMENTATION  IN  IMPROVING  ESTIMATES 

Regardless  of  the  nature  of  the  proposed  AIS,  the  first  step  in  preparing  cost  and 
benefit  estimates  is  to  document  the  baseline  system  and/or  procedures  that  the 
proposed  AIS  will  replace.  Providing  adequate  documentation  of  the  program 
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baseline  is  usually  laborious,  but  it  is  critical  to  the  effective  measurement  of  AIS 
benefits. 

For  the  AISs  we  examined,  program  baseline  documentation  was  based  on  point 
estimates  derived  from  snapshots  of  current  operations  or  from  extrapolations  of 
limited  survey  data.  In  some  cases,  the  data  were  as  much  as  15  years  old.  Although 
the  program  manager  needs  to  expend  resources  judiciously,  investing  in  quality 
baseline  documentation  early  in  the  development  process  can  provide  substantial 
returns  later  in  the  system  life  cycle. 

Problems  resulting  from  poorly  documented  program  baselines  are  exacerbated 
when  the  system  requirements  evolve.  For  example,  in  analyzing  EDCARS,  we 
found  that  the  system  provided  a  substantial  number  of  benefits  that  were  not 
anticipated  in  the  original  system  concept.  Its  principal  benefits  were  that  it 
increased  competitive  procurements.  However,  no  data  were  (or  are)  available  to 
measure  the  value  of  all  the  components  of  that  benefit,  such  as  reductions  in 
contracting  administrative  lead-time  and  reductions  in  the  cost  of  reprocuring  lost  or 
missing  engineering  data.  Moreover,  as  the  EDCARS  hardware  and  software  became 
available,  additional  users  found  applications  for  the  EDCARS  data,  and  additional 
benefits  were  identified.  Because  the  program  baseline  was  not  fully  documented, 
the  benefits  cannot  be  quantified,  and  "re-creation”  of  the  baseline  operating 
environment  is  proving  to  be  difficult  and  expensive. 

This  problem  is  even  more  pronounced  if  the  functional  specifications  of  the 
system  change  over  time:  rebaselining  a  program  to  reflect  new  functions  is  rare. 

A  thorough  evaluation  of  the  program  baseline  is  also  valuable  in  case  some  of 
the  originally  expected  benefits  do  not  materialize.  To  the  extent  that  tangible  and 
quantifiable  benefits  frequently  drive  AIS  requirements,  AISs  are  easier  to  justify  on 
tangible  economic  grounds.  And  since  quantifiable  benefits  are  given  prominence  in 
the  cost-benefit  analyses,  a  well-documented  program  baseline  can  maintain  the 
continued  viability  of  a  program  over  time  by  providing  a  solid  basis  for  establishing 
cost  savings  and  by  broadening  the  base  upon  which  program  benefits  are  justified. 
These  characteristics  become  increasingly  important  to  maintaining  program 
continuity  if  program  costs  increase  beyond  expectations,  if  key  benefits  are  smaller 
than  expected,  or  if  budget  reductions  force  agency- wide  cutbacks. 
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For  example,  in  the  benefit/cost  analysis  of  WSMIS,  nearly  all  projected  benefits 
were  intangible  and  could  not  be  quantified.  Of  those  that  were  tangible,  data  did  not 
exist  to  yield  quantifiable  estimates  for  all  but  one  benefit.  For  that  one  remaining 
benefit,  the  estimate  in  the  economic  analysis  was  projected  to  be  a  "maximum 
benefit,”  with  a  smaller  value  possible.  Since  no  procedures  were  established  to  track 
the  attainment  of  benefits,  the  economic  benefit  realized  by  WSMIS  may  in  fact  be 
substantially  less  than  originally  projected. 

Users  of  WSMIS  report  that  it  has  substantially  increased  their  capabilities. 
That  these  capabilities  provide  intangible  benefits  does  not  diminish  their  value. 
However,  intangible  benefits  may  not  be  as  prominent  in  other  MAISRC-level  AISs. 
Therefore,  there  is  a  need  to  evaluate  tangible  benefits  as  thoroughly  as  possible. 

When  systems  are  justified  on  the  basis  of  quantifiable  economic  criteria,  AIS 
program  managers  need  to  analyze  program  baselines  thoroughly  to  ensure  that 
benefits  as  presented  in  MAISRC  submissions  are  not  understated.  At  the  same 
time,  they  should  not  deemphasize  unquantifiable  system  benefits  simply  because 
economic  criteria  are  sufficient  to  show  a  positive  net  present  value. 

AIS  program  managers  need  to  analyze  system  benefits  more  closely, 
particularly  in  an  environment  of  constrained  funding  or  cost  growth.  Operating 
baselines  must  be  well  defined,  and  adequate  procedures  need  to  be  implemented  to 
track  both  benefits  and  costs. 
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CHAPTER  4 


ORGANIZATIONAL  ISSUES 

COORDINATION  WITH  OTHER  ORGANIZATIONS 

While  the  analysis  of  costs  and  benefits  is  vitally  important  to  the  successful 
preparation  of  a  MAISRC  submission,  each  such  submission  requires  that 
OASD(P&L)  interact  with  numerous  other  OSD  organizations.  During  our  study,  we 
found  that  sensitivity  to  those  interactions  had,  at  times,  been  as  important  to  the 
submitting  organization  as  the  technical  issues  surrounding  cost  analysis. 

Two  OSD  organizations  —  OASD(PA&E)  and  the  Office  of  the  Director, 
Operational  Test  and  Evaluation  (ODOT&E)  —  figured  prominently  during  the 
course  of  our  analysis  of  MAISRC  submissions.  In  fulfilling  their  assigned 
responsibilities,  each  can  significantly  affect  P&L  MAISRC  submissions.  First,  P&L 
needs  to  ensure  early  and  continual  coordination  with  PA&E.  That  coordination  is 
critical  to  the  success  of  the  MAISRC  process.  Second,  P&L  needs  to  design  MAISRC 
submissions  to  meet  the  ongoing  requirements  of  DOT&E  for  comprehensive  test  and 
evaluation.  By  understanding  the  requirements  and  perspectives  of  these 
organizations,  P&L  can  structure  MAISRC  presentations  and  procedures  for 
maximum  effectiveness. 

PA&E  PERSPECTIVE 

PA&E  contends  that  cost  and  benefit  submissions  to  the  MAISRC  should 
provide  two  key  elements:  adequate  program  definition  and  adequate 
documentation.  If  the  MAISRC  submission  contains  these  elements,  P&L  and  PA&E 
will  have  a  common  basis  for  agreement  on  cost  and  benefit  estimates.  By  addressing 
these  concerns  early  in  the  MAISRC  process,  P&L  can  minimize  the  delay  caused  by 
PA&E  review. 

Program  Definition 

The  PA&E  role  in  preparation  of  submissions  to  MAISRC  is  to  validate  cost  and 
benefit  estimates;  MAISRC  does  not  require  it  to  prepare  an  independent  cost 
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estimate  (ICE).  PA&E  does  not  merely  review  the  submission;  it  attempts  to 
understand  and  validate  the  analysis.  The  Comptroller  has  provided  instructions 
Department  of  Defense  Instruction  [(DoDI)  7041.3,  Economic  Analysis  and  Program 
Evaluation  for  Resource  Management,  and  DoDI  7920.2,  Automated  Information 
System  ( AIS)  Life-Cycle  Management  and  Milestone  Approval  Procedures]  on  how  to 
conduct  economic  analyses.  These  regulations  serve  as  guidelines  for  the  Services  in 
preparing  economic  analyses  for  MAISRC  submissions,  and  they  provide 
recommended  lead-times  to  ensure  timely  review. 

In  validating  the  estimates,  PA&E  must  first  understand  the  requirement  the 
Service  is  trying  to  fulfill.  To  do  so  it  consults  the  program  office  for  a  definition  of  its 
mission.  It  then  subdivides  the  mission  into  functional  areas  and  maps  the  status  quo 
system  into  those  functional  areas. 

Having  defined  the  status  quo,  PA&E  then  defines  the  "program”  by  stating  the 
objectives  of  the  program,  how  those  objectives  relate  to  the  mission,  and  how  the 
program  will  affect  the  status  quo.  PA&E  is  particularly  concerned  with  specifying 
which  activities  are  classified  as  maintenance  of  the  old  system  and  which  are 
classified  as  improvements  to  the  old  system.  As  a  final  step,  PA&E  verifies  that  the 
Services  have  defined  a  realistic  schedule  for  meeting  the  program  objectives. 

By  performing  these  actions  —  defining  the  status  quo,  defining  the  program, 
and  verifying  the  realism  of  the  program  schedule  —  before  PA&E  does,  P&L  can 
effectively  design  a  program  to  develop  and  implement  an  AIS.  The  MAISRC 
submission  can  then  readily  be  derived  from  this  design. 

Program  Documentation 

The  principal  problem  with  MAISRC  submissions,  as  reported  by  PA&E,  is 
documentation.  PA&E  will  not  permit  a  MAISRC  submission  to  go  forward  until  the 
documentation  is  in  order.  Fundamentally,  good  documentation  requires  a 
well-conceived  plan  for  meeting  the  program  objectives,  in  as  much  detail  as  possible. 
In  its  validation  role,  PA&E  places  a  strong  emphasis  on  both  logic  and  believability 
in  AIS  estimates.  Documentation  plays  a  key  role  in  meeting  these  criteria. 

Good  documentation  invariably  accelerates  the  MAISRC  review  process. 
Although  PA&E  may  not  agree  with  the  Service’s  estimate,  clear  documentation  of 
the  estimate  at  least  provides  a  well-defined  basis  for  discussion.  Preferably,  such 
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documentation  should  be  provided  to  PA&E  as  early  in  the  process  as  possible.  By 
submitting  a  "straw  man”  cost  estimate  early,  the  Services  provide  the  opportunity 
for  early  clarification  and  correction,  thereby  facilitating  MAISRC  approval. 

DOT&E  PERSPECTIVE 

Submissions  to  the  MAISRC  must  include  plans  for  testing  that  is  sufficient  to 
demonstrate  the  efficacy  of  the  system.  The  plans  are  reviewed  by  DOT&E,  and  two 
essential  elements  of  that  review  are  assessments  of  the  sponsor’s  test  and  evaluation 
master  plan  (TEMP)  and  the  operational  test  plan.  Organizations  sponsoring 
MAISRC  submissions  need  to  devote  careful  attention  to  test  plans  in  general  and  to 
the  TEMP  in  particular. 

At  Milestone  I,  DOT&E’s  principal  concern  is  to  verify  that  the  System  Concept 
Paper  (SCP)  adequately  addresses  testing.  In  particular,  it  is  concerned  about  the 
TEMP,  which  needs  to  be  included  in  MAISRC  submissions  beginning  with  the 
earliest  milestones.  The  TEMP  is  divided  into  five  parts: 

•  System  Description  —  This  section  defines  the  risks  involved  in  developing 
the  AIS  and  delineates  how  the  proposed  tests  will  diminish  those  risks. 

•  Management  —  This  section  describes  who  will  manage  the  program  and 
where  and  when  the  testing  will  be  accomplished.  More  important,  it 
describes  who  the  independent  testers  will  be  and  who  the 
"non-independent”  testers  will  be.  This  particular  aspect  is  critical  to 
DOT&E  because  it  needs  to  ascertain  that  the  test  team  is  independent  of 
the  program  management. 

•  Tests  —  This  section  describes  the  tests  that  will  be  performed  on  the  system 
by  people  reporting  to  the  program  manager.  Descriptions  of  the  tests  need 
to  be  specific. 

•  Independent  Test  Plan  —  This  section  describes  the  testing  that  will  be  done 
by  someone  other  than  the  program  manager  (usually  a  peer  or  higher 
independent  authority). 

•  Resources  —  This  section  describes  the  time,  money,  and  personnel 
necessary  to  conduct  testing.  Explicit  recognition  of  resource  ccsts  is  desired 
since  test  costs  are  frequently  embedded  in  other  elements  of  the  MAISRC 
submission.  DOT&E  will  ensure  that  cost  estimates  are  consistent  with  the 
time  estimates,  and  that  both  are  reasonable  compared  to  previous 
experience.  Deviations  need  to  be  explained. 
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Major  Issues  Concerning  the  Content  of  the  TEMP 

From  the  DOT&E  perspective,  the  principal  difficulties  in  gaining  MAISRC 
approval  arise  from  the  Independent  Test  Plan  section  of  the  TEMP.  For  example, 
the  Services  need  to  define  "operational  tests”  adequately  and  determine  what  the 
"critical  issues”  are.  These  issues  need  to  be  resolved  in  concert  with  the  user 
community,  which  frequently  has  not  adequately  considered  them  prior  to  Milestone 
I.  Consequently,  if  P&L  proposes  imprecise  tests  that  may  not  validate  the  utility  of 
the  system,  then  DOT&E  will  reject  the  MAISRC  submission. 

Another  problem  arises  from  determining  how  long  the  tests  should  run. 
DOT&E  recommends  they  run  at  least  a  month,  but  shorter  or  longer  periods  can  be 
justified  in  the  TEMP  if  they  are  shown  to  be  reasonable.  The  objective  of  the  tests 
should  be  to  demonstrate  system  capability  under  a  variety  of  actual  operating 
conditions.  To  the  maximum  extent  possible,  P&L  needs  to  anticipate  the  full  range 
of  expected  conditions  and  plan  tests  accordingly. 

DOT&E  recommends  that  actual  data  be  used  in  the  tests,  but  if  the  full  range 
of  conditions  is  not  expected  during  the  test  window  fe.g.,  heavy  end-of-year 
processing),  then  conditions  for  the  test  may  be  simulated.  The  General  Accounting 
Office,  in  particular,  has  expressed  concern  that  systems  be  operational  during 
periods  of  stress,  such  as  would  occur  in  wartime.  If  the  test  is  simulated,  then  for 
test  purposes,  such  stress  conditions  should  also  be  simulated. 

Development  tests  on  system  components  are  acceptable  substitutes  for 
operational  tests.  DOT&E  encourages  development  tests  because  of  congressional 
concern  over  the  level  of  system  implementation  required  to  carry  out  operational 
tests.  For  large  systems,  operational  tests  can  be  expensive,  and  any  shortcomings 
detected  during  those  tests  are  more  expensive  to  correct  than  they  would  be  if 
detected  during  earlier  development  testing.  P&L  should  therefore  consider 
including  development  tests  in  lieu  of  operational  tests  in  their  MAISRC 
submissions. 

Other  Issues  in  DOT&E's  Review  of  MAISRC  Submissions 

DOT&E  looks  for  four  major  characteristics  in  its  evaluation  of  MAISRC 
submissions:  test  team  independence,  adequate  resources  for  testing,  robustness  of 
the  testing,  and  training. 
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The  independence  of  the  "independent”  test  team  is  especially  problematic  for 
smaller  agencies,  such  as  The  Defense  Logistics  Agency  (DLA),  where  the  proposed 
independent  testers  may  actually  have  a  close  (or  subordinate)  relationship  to  the 
program  manager.  MAISRC  submissions  need  to  clearly  state  how  independence  will 
be  achieved. 

DOT&E  is  also  very  concerned  that  the  resources  allocated  for  testing  are 
adequate.  Frequently,  MAISRC  rejects  submissions  because  neither  enough  time  nor 
enough  funding  is  allotted  to  testing. 

The  Program  Office  also  needs  to  design  the  TEMP  to  ensure  that  enough  of  the 
system  will  be  tested  to  yield  justifiable  results.  "Justifiable  results”  implies  that  the 
tests  should  accommodate  a  variety  of  operating  conditions.  Of  particular  interest  to 
DOT&E  is  the  discussion  of  what  will  not  be  tested  and  what  cannot  be  tested.  The 
TEMP  should  discuss  the  effect  of  any  omissions  on  the  independent  tests  (Part  4)  and 
on  the  program  manager’s  knowledge.  The  increased  risks  should  be  delineated,  and 
the  TEMP  should  describe  the  actions  the  program  manager  intends  to  take  to 
mitigate  those  risks. 

Finally,  the  TEMP  needs  to  address  training.  Training  has  been  characterized 
by  DOT&E  as  the  biggest  single  impediment  to  system  utilization.  P&L’s  MAISRC 
submissions  should  therefore  provide  sufficient  training  plans  to  enhance  the 
effectiveness  of  the  AIS.  Documentation  of  these  training  plans  in  the  MAISRC 
submission  will  increase  the  likelihood  of  MAISRC  approval. 
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CHAPTER 5 


CONCLUSIONS  AND  RECOMMENDATIONS 


Standardizing  the  development  and  content  of  OASD(P&L)  submissions  to  the 
MAISRC  can  reduce  delays  in  AIS  development.  Areas  that  can  benefit  most  from 
standardization  include:  benefits  analysis,  cost  analysis,  and  organizational 
interfaces.  Tangible,  quantifiable  AIS  benefits  generally  provide  the  economic 
justification  for  an  investment  while  intangible  AIS  benefits,  which  can  also  be 
substantial,  provide  additional  program  justification  in  an  environment  of  cost 
growth  or  constrained  funding.  P&L  should  increase  its  emphasis  on  documenting 
intangible  AIS  benefits  and  examine  innovative  techniques  for  assessing  the  value  of 
such  benefits. 

For  an  accurate  estimation  and  analysis  of  benefits  and  costs,  P&L  must  clearly 
identify  the  program  baseline.  Failure  to  document  the  program  baseline  adequately 
makes  cost  estimates  questionable  and  severely  hampers  the  validation  of  benefits 
after  the  system  is  implemented.  Changes  in  functional  requirements  or  the  addition 
of  new  users  to  the  system  can  only  be  evaluated  against  a  thoroughly  documented 
baseline.  P&L  should  devote  additional  effort  to  ensuring  adequate  baseline 
documentation. 

With  respect  to  costs,  P&L  needs  to  ensure  that  the  proposed  cost  structure  is 
consistent  with  the  capability  being  proposed.  All  capabilities  should  have 
corresponding  costs,  and  those  costs  should  be  tabulated  across  the  full  system  life 
cycle.  P&L  should  precisely  articulate  system  requirements  and  structure  the 
proposed  cost  analysis  to  reflect  these  requirements.  P&L  needs  to  clearly  document 
the  program  costs  and  benefits  and  indicate  how  each  was  measured. 

P&L  should  encourage  the  use  of  parametric  costing  methods  at  an  early  stage 
to  supplement  analogy-based  procedures  and  to  validate  engineering  estimates  and 
vendor  quotes.  As  a  first  step  in  developing  appropriate  parametric  cost  estimating 
tools,  P&L  should  create  a  database  of  historical  costs  of  P&L-sponsored  AISs. 
Special  problem  areas  for  P&L-sponsored  AISs,  such  as  software  development  and 
maintenance,  should  be  given  first  priority. 
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Organizationally,  P&L  can  take  several  steps  to  expedite  the  MAISRC  approval 
process.  Early  and  frequent  communications  with  other  organizations  in  the 
MAISRC  process  —  PA&E  and  DOT&E  are  two  —  are  essential.  Consistency  in 
presentation  format  with  PA&E  guidance  is  desirable  although  as  a  practical  matter 
such  consistency  may  not  be  attainable. 
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PA&E  LIFE-CYCLE  COST  ELEMENTS 


Source;  "Department  of  Defense  Automated  Information  System  Life  Cycle  Cost 
and  Benefits  Estimation  Guide  (DoD  Cost/Benefits  Guide),”  Office  of  the  Assistant 
Secretary  of  Defense,  Program  Analysis  and  Evaluation,  30  May  1989. 
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PA&E  LIFE-CYCLE  COST  ELEMENTS  (cont.) 


PA&E  guidelines 

2.02 

Software 

2.021 

Commercial  off-the-shelf  (COTS) 

2.0211 

Operating  system  software 

2.0212 

General  administrative  software 

2.022 

Mission-specific  application  software 

2.0221 

Contractor-developed  software 

2.0222 

Organically  developed  software 

2.023 

Communications  software 

2.0231 

Wide-area  gateways  (Broadband) 

2.0232 

Wide-area  networks 

2.0233 

Modems 

2.0234 

Local  area  networks 

2.0235 

Crypto  software 

2.0236 

Other  software 

2.04 

Systems  integration,  testing,  and  evaluation 

2.05 

Program  management 

2.06 

Training 

2.07 

Support  equipment 

2.08 

Initial  spares 

2.09 

Initial  cataloging 

2.10 

Initial  data  requirements 

2.11 

Site  activation 

2.12 

Industrial  facilities 

2.13 

Warranties 

2.14 

Initial  supplies 

2.15 

Engineering  changes 

Source;  'Department  of  Defense  Automated  Information  System  Life  Cycle  Cost 
and  Benefits  Estimation  Guide  (DoD  Cost/Benefits  Guide),'  Office  of  the  Assistant 
Secretary  of  Defense,  Program  Analysis  and  Evaluation,  30  May  1989. 
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PA&E  LIFE-CYCLE  COST  ELEMENTS  (cont.) 


PA&E  guidelines 


2.16 

2.161 

2.162 

2.17 

2.18 

3.0 

3.01 

3.011 

3.012 

3.014 

3.02 

3.03 

3.031 

3.032 

3.04 

3.05 

3.06 

3.07 

3.071 

3.072 

3.073 

3.08 

3.081 

3.082 

3.09 

4.0 


Preplanned  product  improvements 

Hardware  preplanned  product  improvements 
Software  preplanned  product  improvements 
Upgrades 

Offices  and  general  support  furniture 

OPERATING  AND  SUPPORT 

System/material/item  management 
System  management 
Operating  personnel 
Training 

Unit/base  operations 
Hardware  maintenance  support 
Depot  level 
Field  level 

Second  destination  transportation 

Environmental  and  hazardous  material  storage  and  handling 
Contract  leasing 

Operations  investment 
Replenishment  spares 

Fuel  and  petroleum,  oil,  and  lubricants  (POL) 

Replenishable  supplies  and  consumables 

Software  maintenance 

Central  maintenance  and  repair,  software 
Field  operation  maintenance,  software 
Parallel  system  operation 

DISPOSAL 


Source:  'Department  of  Oefense  Automated  Information  System  life  Cycle  Cost  and  Benefits 
Estimation  Guide  (DoD  Cost/Benefits  Guide),"  Office  of  the  Assistant  Secretary  of  Defense, 
Program  Analysis  and  Evaluation,  30  May  1989 


COST  AND  BENEFIT  ANALYSIS 
OASD(PA&E)-RECOMMENDED  FORMATS 


FORMAT  A 

COMPARISON  WITH  PREVIOUS  ESTIMATE 

(FY _ $  in  millions) 

(Total  program  or  alternative) 


Element 

no.a 

Cost  element  title 

Current 
est.  date 

Previous 
est.  date 

Delta 

Note 

1.0 

RDT&E 

XX 

XX 

X 

b 

1.01 

Program  planning 
and  management 

1.02 

Development  hardware 

2.0 

Investment 

XX 

XX 

X 

b 

2.01 

Hardware 

2.02 

Software 

3.0 

Operating  and  support 

XX 

XX 

3.01 

System/material/item 

management 

3.02 

Unit/base  operations 

_ 

Note:  OASD(PASiE)  »  Office  of  the  Assistant  Secretary  of  Defense  (Program  Analysis  and  Evaluation);  RDT&E  =*  research, 
development,  test  and  evaluation. 

Reference  Appendix  A. 
bExplanation  of  delta. 
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FORMAT  B 


PRIOR  YEAR  COSTS 

(FY _ $  in  millions) 

(Total  program  or  alternative) 


Element 

no.* 

Cost  element  title 

FY 

Total 

RDT&E 

FY-N,  FY-4,  FY-3,  FY-2,  FY-1 

;  IPTf 

Program  planning  and  management 

IK" K » j;l| 

Development  hardware 

Investment 

2.01 

Hardware 

2.02 

Software 

3.0 

Operating  and  support 

3.01 

System/material/item  management 

3.02 

Unit/base  operations 

Totals 

Reference  Appendix  A. 


Program  start  date _ 

Program  completion  date _ 

Date  of  pending  MAISRC  review  _ 

Instruction:  Display  cost  from  the  first  year  of  program  inception  to  current 
fiscal  year  for  each  alternative  or  the  total  program  as  appropriate.  Where  possible 
use  the  same  cost  element  structure  as  Format  A. 


FORMAT  C 


LIFE-CYCLE  COST  ESTIMATE 

(FY _ $  in  millions) 

(Total  program  or  alternative) 


Element 

no.* 

Cost  element  title 

Prior  years 
FYXX  toFYXX 

FYXX 

FY-n 

Total 

1.0 

RDT&E 

1.01 

Program  planning  and 
management 

1.02 

Development  hardware 

2.0 

Investment 

2.01 

Hardware 

2.02 

Software 

3.0 

Operating/support 

3.01 

System/material/item 

management 

3.02 

Unit/base  operations 

Totals 

^Reference  Appendix  A. 


Instructions:  I. 

2. 

3. 


Notes  will  be  numbered  to  correspond  with  element  numbers. 
Cover  10  years  after  year  of  full  operational  capability. 

Cover  upgrade  caused  by  wearing  out  and  obsolescence. 


4.  Include  prior  year  in  total. 
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FORMAT  D 


PM  AND  ICE  COMPARISON 

(FY _ $  in  millions) 

(Total  program  or  alternative) 


Element 

no.a 

Cost  element  title 

PM 

estimate 

ICE 

estimate 

Delta 

% 

Delta 

1.0 

RDT&E 

1.01 

Program  planning  and 

management 

1.02 

Development  hardware 

2.0 

Investment 

2.01 

Hardware 

2.02 

Software 

3.0 

Operating  and  support 

3.01 

System/material/item 

management 

3.02 

Unit/base  operations 

Totals 

Mote;  PM  »  program  manager;  ICE  a  independent  cost  estimate. 
“Reference  Appendix  A. 
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FORMATE 


PROGRAM  REQUIREMENTS  &  POM  OR  BUDGET  COMPARISON 
(Then-year  $  in  millions) 


FYXX 

FYXX 

FYXX 

FYXX 

FYXX 

FYXX 

Requirement 

R&D 

Investment 

o&s 

Total 

Last  POM  or  budget  submission 
as  appropriate* 

R&D 

Investment 

O&S 

Total 

Excess/(Shortfall) 

R&D 

Investment 

O&S 

Total 

Note:  POM  =■  Program  Objective  Memorandum;  O&S  =  operations  and  support, 
identify  document. 
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FORMAT  F 


COMPARISON  OF  ALTERNATIVE  WITH  STATUS  QUO 

(FY _ S  in  millions) 

Alternative  title: 


Element 

no.a 

Cost  element  title 

Current 

FYXX 

FYXX 

FY-N 

Total 

Alt.** 

SQC 

Alt.b 

SQC 

Alt.b 

SQ< 

Alt.b 

SQ« 

1.0 

RDT&E 

1.01 

Program  planning  and 
management 

1.02 

Development  hardware 

2.0 

Investment 

2.01 

Hardware 

2.02 

Software 

3.0 

Operating  and  support 

3.01 

System/material/item 

management 

3.02 

Unit/base  operations 

Totals 

^Reference  Appendix  A 

bFor  the  alternative  (Alt.)  named  in  the  format  title. 
‘Cost  for  the  status  quo  (SQ)  alternative. 
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FORMAT  G 


COMPARISON  WITH  PREVIOUS  QUANTIFIABLE  BENEFIT  ESTIMATE 

(FY _ $  in  millions) 

(Total  program  or  alternative) 


Element 

no.« 

Cost  element  title 

Current 
est.  date 

Previous 
est.  date 

Delta 

Note 

1.0 

RDT&E 

XX 

XX 

X 

1 

1.01 

Program  planning 
and  management 

1.02 

Development  hardware 

2.0 

Investment 

XX 

XX 

X 

2 

2.01 

Hardware 

XX 

XX 

X 

3 

2.02 

Software 

3.0 

Operating  and  support 

XX 

XX 

X 

4 

3.01 

System/material/item 

management 

3.02 

Unit/base  operations 

Totals 

^Reference  Appendix  A. 
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FORMAT  H 


QUANTIFIABLE  BENEFIT  ESTIMATE 

(FY _ $  in  millions) 

(Total  program  or  alternative) 


Element 

no.* 


Cost  element  title 


RDT&E 

Program  planning  and 
management 

Development  hardware 
Investment 

Hardware 

Software 

Operating/support 

System/material/item 

management 

Unit/base  operations 


Totals 


^Reference  Appendix  A. 

Instructions:  1.  Explanatory  notes,  if  nettled,  will  be  uuxnbered  to  correspond 
with  element  numbers. 

2.  Cover  10  years  after  year  of  full  operational  capability. 


3.  Cover  upgrade  caused  by  wearing  out  and  obsolescence. 
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